Improving the validity of Script Concordance Testing by optimising and balancing items

Background: A script concordance test (SCT) is a modality for assessing clinical reasoning. Concerns had been raised about the plausible validity threat to SCT scores if students deliberately avoided the extreme answer options to obtain higher scores. The aims of the study were firstly to investigate whether students’ avoidance of the extreme answer options could result in higher scores, and secondly to determine whether a ‘balanced approach’ by careful construction of SCT items (to include extreme as well as median options as model responses) would improve the validity of an SCT. Methods: Using the paired sample t-test, the actual average student scores for 10 SCT papers from 2012–2016 were compared with simulated scores. The latter were generated by recoding all ‘2’ responses to ‘1’ and ‘+2’ responses to ‘+1’ for the whole and bottom 10% of the cohort (simulation 1), and scoring as if all students had chosen ‘0’ for their responses (simulation 2). The actual average and simulated average scores in 2012 (before the ‘balanced approach’) were compared with those from 2013–2016, when papers had a good balance of modal responses from the expert reference panel. Results: In 2012, a score increase was seen in simulation 1 in the third-year cohort, from 50.2 to 55.6% (t [10] = 4.818; p = 0.001). Since 2013, with the ‘balanced approach’, the actual SCT scores (57.4%) were significantly higher than scores in both simulation 1 and simulation 2 (46.7% and 23.9% respectively). Conclusions: When constructing SCT examinations, apart from the rigorous pre-examination optimisation, it is desirable to achieve a balance between items that attract extreme responses and those that attract median response options. This could mitigate the validity threat to SCT scores, especially for the low-performing students who have previously been shown to only select median responses and avoid the extreme responses

The Telehealth Skills, Training, and Implementation Project: An evaluation protocol

External stabilization is reported to improve reliability of hand held dynamometry, yet this has not been tested in burns. We aimed to assess the reliability of dynamometry using an external system of stabilization in people with moderate burn injury and explore construct validity of strength assessment using dynamometry. Participants were assessed on muscle and grip strength three times on each side. Assessment occurred three times per week for up to four weeks. Within session reliability was assessed using intraclass correlations calculated for within session data grouped prior to surgery, immediately after surgery and in the sub-acute phase of injury. Minimum detectable differences were also calculated. In the same timeframe categories, construct validity was explored using regression analysis incorporating burn severity and demographic characteristics. Thirty-eight participants with total burn surface area 5 – 40% were recruited. Reliability was determined to be clinically applicable for the assessment method (intraclass correlation coefficient \u3e0.75) at all phases after injury. Muscle strength was associated with sex and burn location during injury and wound healing. Burn size in the immediate period after surgery and age in the sub-acute phase of injury were also associated with muscle strength assessment results. Hand held dynamometry is a reliable assessment tool for evaluating within session muscle strength in the acute and sub-acute phase of injury in burns up to 40% total burn surface area. External stabilization may assist to eliminate reliability issues related to patient and assessor strength

The influence of longitudinal integrated clerkships on clinical scholarship in regional and rural communities of practice

All senior medical students at the University of Wollongong undertake a longitudinal integrated community-based cle rkship, supervised by experienced generalist practitioners (GPs) in a regional or ru ral community. Prior to the first clerkship, GP preceptors predicted outcomes from their relationship with the long-term students, including a more scholarly approach to clinical pract ice. This prompted exploration of preceptor post·clerkship perceptions In the light of Grigsby and Tl10rndykes (2011) recent working definition of dlnical scholarship in academic medicine

Longitudinal integrated clerkships^*

<p>There is increased interest in longitudinal integrated clerkships (LICs) due to mounting evidence of positive outcomes for students, patients and supervising clinicians. Emphasizing continuity as the organizing principle of an LIC, this article reviews evidence and presents perspectives of LIC participants concerning continuity of care, supervision and curriculum, and continuity with peers and systems of care. It also offers advice on implementing or evaluating existing LIC programs.</p

Improving assessment practice through cross-institutional collaboration: an exercise on the use of OSCEs

Background: This study was undertaken to improve assessment practice on OSCEs through collaboration across geographically dispersed medical schools in Australia.\ud \ud Methods: A total of eleven OSCE stations were co-developed by four medical schools and used in summative 2011 and 2012 examinations for the assessment of clinical performance in the early clinical and exit OSCEs in each school's medical course. Partial Credit Rasch Model was used to evaluate the psychometric properties of the shared OSCE data. Evaluation of the quality assurance reports was used to determine the beneficial impact of the collaborative benchmarking exercise on learning and teaching outcomes.\ud \ud Results: The data for each examination demonstrated sufficient fit to the Rasch model with infit mean square values ranging from 0.88 to 0.99. Person separation (1.25–1.63) indices indicated good reliability. Evaluation of perceived benefits showed that the benchmarking process was successful as it highlighted common curriculum areas requiring specific focus and provided comparable data on the quality of teaching at the participating medical schools.\ud \ud Conclusion: This research demonstrates the validity of the psychometric data and benefits of evaluating clinical competence across medical schools without the enforcement of a prescriptive national curriculum or assessment

The Telehealth Skills, Training, and Implementation Project: An Evaluation Protocol

Background: Telehealth appears to be an ideal mechanism for assisting rural patients and doctors and medical students/registrars in accessing specialist services. Telehealth is the use of enhanced broadband technology to provide telemedicine and education over distance. It provides accessible support to rural primary care providers and medical educators. A telehealth consultation is where a patient at a general practice, with the assistance of the general practitioner or practice nurse, undertakes a consultation by videoconference with a specialist located elsewhere. Multiple benefits of telehealth consulting have been reported, particularly those relevant to rural patients and health care providers. However there is a paucity of research on the benefits of telehealth to medical education and learning. Objective: This protocol explains in depth the process that will be undertaken by a collaborative group of universities and training providers in this unique project. Methods: Training sessions in telehealth consulting will be provided for participating practices and students. The trial will then use telehealth consulting as a real-patient learning experience for students, general practitioner trainees, general practitioner preceptors, and trainees. Results: Results will be available when the trial has been completed in 2015. Conclusions: The protocol has been written to reflect the overarching premise that, by building virtual communities of practice with users of telehealth in medical education, a more sustainable and rigorous model can be developed. The Telehealth Skills Training and Implementation Project will implement and evaluate a theoretically driven model of Internet-facilitated medical education for vertically integrated, community-based learning environment